Combining Nearest Neighbor Classifiers Through Multiple Feature Subsets

Combining multiple classi ers is an e ective technique for improving accuracy. There are many general combining algorithms, such as Bagging or Error Correcting Output Coding, that signi cantly improve classi ers like decision trees, rule learners, or neural networks. Unfortunately, many combining methods do not improve the nearest neighbor classi er. In this paper, we present MFS, a combining algorithm designed to improve the accuracy of the nearest neighbor (NN) classi er. MFS combines multiple NN classi ers each using only a random subset of features. The experimental results are encouraging: On 25 datasets from the UCI Repository, MFS signi cantly improved upon the NN, k nearest neighbor (kNN), and NN classi ers with forward and backward selection of features. MFS was also robust to corruption by irrelevant features compared to the kNN classi er. Finally, we show that MFS is able to reduce both bias and variance components of error.